Skill-Aware Task Assignment in Crowdsourcing Applications

International audienceBesides simple human intelligence tasks such as image labeling, crowdsourcing platforms propose more and more tasks that require very specific skills. In such a setting we need to model skills that are required to execute a particular job. At the same time in order to match tasks to the crowd, we have to model the expertise of the participants. We present such a skill model that relies on a taxonomy. We also introduce task assignment algorithms to optimize the result quality. We illustrate the effectiveness of our algorithms and models through preliminary experiments with synthetic datasets

Identifying potentially flawed items in the context of small sample IRT analysis

Although Classical Test Theory has been used by the measurement community for almost a century, Item Response Theory has become commonplace for educational assessment development, evaluation and refinement in recent decades. Its potential for improving test items as well as eliminating the ambiguous or misleading ones is substantial. However, in order to estimate its parameters and produce reliable results, IRT requires a large sample size of examinees, thus limiting its use to large-scale testing programs. Nevertheless, the accuracy of parameter estimates becomes of lesser importance when trying to detect items whose parameters exceed a threshold value. Under this consideration, the present study investigates the application of IRT-based assessment evaluation to small sample sizes through a series of simulations. Additionally, it introduces a set of quality indices, which exhibit the success rate of identifying potentially flawed items in a way that test developers without a significant statistical background can easily comprehend and utilize

Ανάπτυξη λογισμικού σε περιβάλλον ROS για την κίνηση βραχιόνων ρομπότ διαστημικού εξομοιωτή και σχεδιασμός τροχιάς για την αυτόματη σύλληψη στόχων.

Η παρούσα πτυχιακή εργασία, έχει ως σκοπό την επίλυση κάποιων προβλημάτων στο λογισμικό του ενεργητικού ρομπότ (Cepheus) του διαστημικού εξομοιωτή του εργαστηρίου, καθώς επίσης την αναβάθμιση της λειτουργικότητας του με την προσθήκη νέου hardware και νέων υλοποιήσεων στο λογισμικό. Όσον αφορά το λογισμικό, μειώθηκαν κάποια scheduling latencies σε διεργασίες του συστήματος του ρομπότ, με τη χρήση real-time scheduling policies. Επίσης, οργανώθηκε με σωστότερο τρόπο η εκκίνηση των controllers στο σύστημα του ρομπότ, με την ανάπτυξη κώδικα στο ROS. Επίσης, προστέθηκαν στους βραχίονες του ρομπότ δύο νέοι κινητήρες αρθρώσεων καρπού και αρπάγης και διασυνδέθηκαν κατάλληλα με το λογισμικό. Στη συνέχεια, τοποθετήθηκαν κατάλληλοι αισθητήρες δύναμης στις αρπάγες και υλοποιήθηκε έλεγχος δύναμης (force control) σε αυτές με ανάδραση από τους αισθητήρες. Επίσης, υλοποιήθηκε στο ROS, η διαδικασία της αντίστροφης κινηματικής για τους βραχίονες του ρομπότ, με σκοπό τη σύλληψη αντικειμένων με χρήση των νέων αρθρώσεων. Τέλος, υλοποιήθηκε η μελέτη ενός αλγορίθμου σχεδιασμού τροχιάς για τη βάση του ρομπότ, με σκοπό τη σύλληψη κινούμενων στόχων με αυτόματο τρόπο. Ακολούθησε μια προσομοίωση του αλγορίθμου σε Matlab και στη συνέχεια η υλοποίηση του στο ROS. Προς επιβεβαίωση κάποιων περιπτώσεων του αλγορίθμου, πραγματοποιήθηκαν κάποια πειράματα πάνω στο τραπέζι του διαστημικού εξομοιωτή του εργαστηρίου.The goal of this thesis is to solve a number of problems related to the software of the active robot (Cepheus) of the space emulator, as well as to upgrade the robot’s hardware and software with new additions. About the improvements to the software, the use of the Linux real-time scheduling policies in the robot’s processes, led to a reduction in some significant scheduling latencies. Also, we managed to obtain a more organized way of the initialization of the robot’s controllers, with the development of a ROS process that handles this matter. The robot’s hardware was upgraded with the addition of two wrist joints kai two gripper joints, that were easily connected to the robot’s software for control. Following that, the gripper joints were upgraded with the addition of two force sensors, that were used in the implementation of force control on the grippers. Finally, the ROS code was updated with the implementation of inverse kinematics for the robot’s arms, that gave them the capability of reaching and grabbing objects. The last part of this thesis is about the creation of a trajectory planning algorithm for the robot’s base, that will lead to the chase and catch of moving space targets. Firstly, a mathematical study led to the basics of the algorithm, followed by a Matlab simulation. After that, the algorithm was implemented on the robot’s ROS software. The evaluation, of a number of the algorithm’s concepts, was implemented by the execution of some experiments on the granite table of the space emulator of the laboratory

Macrophage activation syndrome in a child with unclassified systemic vasculitis probably triggered by Parvovirus B19 infection

Macrophage activation syndrome (MAS) is a life threatening complication of chronic rheumatic diseases of childhood and especially of systemic juvenile idiopathic arthritis. Infections, particularly viral, have been suggested to play a triggering role. We describe a case of systemic unclassified ANCA positive vasculitis complicated with fatal MAS triggered probably by Parvovirus B19 infection

Extending LMS to Support IRT-Based Assessment Test Calibration

Developing unambiguous and challenging assessment material for measuring educational attainment is a time-consuming, labor-intensive process. As a result Computer Aided Assessment (CAA) tools are becoming widely adopted in academic environments in an effort to improve the assessment quality and deliver reliable results of examinee performance. This paper introduces a methodological and architectural framework which embeds a CAA tool in a Learning Management System (LMS) so as to assist test developers in refining items to constitute assessment tests. An Item Response Theory (IRT) based analysis is applied to a dynamic assessment profile provided by the LMS. Test developers define a set of validity rules for the statistical indices given by the IRT analysis. By applying those rules, the LMS can detect items with various discrepancies which are then flagged for review of their content. Repeatedly executing the aforementioned procedure can improve the overall efficiency of the testing process

A Survey of Crowdsourcing in Medical Image Analysis

Rapid advances in image processing capabilities have been seen across many domains, fostered by the application of machine learning algorithms to "big-data". However, within the realm of medical image analysis, advances have been curtailed, in part, due to the limited availability of large-scale, well-annotated datasets. One of the main reasons for this is the high cost often associated with producing large amounts of high-quality meta-data. Recently, there has been growing interest in the application of crowdsourcing for this purpose; a technique that a technique that is well established in a number of disciplines, including astronomy, ecology and meteorology for creating large-scale datasets across a range of disciplines, from computer vision to astrophysics. Despite the growing popularity of this approach, there has not yet been a comprehensive literature review to provide guidance to researchers considering using crowdsourcing methodologies in their own medical imaging analysis. In this survey, we review studies applying crowdsourcing to the analysis of medical images, published prior to July 2018. We identify common approaches and challenges and provide recommendations to researchers implementing crowdsourcing for medical imaging tasks. Finally, we discuss future opportunities for development within this emerging domain

Utilisation d'une hiérarchie de compétences pour l'optimisation de sélection de tâches en crowdsourcing

A large number of commercial and academic participative applications rely on a crowd to acquire, disambiguate and clean data. These participative applications are widely known as crowdsourcing platforms where amateur enthusiasts are involved in real scientific or commercial projects. Requesters are outsourcing tasks by posting them on online commercial crowdsourcing platforms such as Amazon MTurk or Crowdflower. There, online participants select and perform these tasks, called microtasks, accepting a micropayment in return. These platforms face challenges such as reassuring the quality of the acquired answers, assisting participants to find relevant and interesting tasks, leveraging expert skills among the crowd, meeting tasks' deadlines and satisfying participants that will happily perform more tasks. However, related work mainly focuses on modeling skills as keywords to improve quality, in this work we formalize skills with the use a hierarchical structure, a taxonomy, that can inherently provide with a natural way to substitute tasks with similar skills. It also takes advantage of the whole crowd workforce. With extensive synthetic and real datasets, we show that there is a significant improvement in quality when someone considers a hierarchical structure of skills instead of pure keywords. On the other hand, we extend our work to study the impact of a participant’s choice given a list of tasks. While our previous solution focused on improving an overall one-to-one matching for tasks and participants we examine how participants can choose from a ranked list of tasks. Selecting from an enormous list of tasks can be challenging and time consuming and has been proved to affect the quality of answers to crowdsourcing platforms. Existing related work concerning crowdsourcing does not use either a taxonomy or ranking methods, that exist in other similar domains, to assist participants. We propose a new model that takes advantage of the diversity of the parcipant's skills and proposes him a smart list of tasks, taking into account their deadlines as well. To the best of our knowledge, we are the first to combine the deadlines of tasks into an urgency metric with the task proposition for knowledge-intensive crowdsourcing. Our extensive synthetic and real experimentation show that we can meet deadlines, get high quality answers, keep the interest of participants while giving them a choice of well selected tasks.Des nombreuses applications participatives, commerciales et académiques se appuient sur des volontaires ("la foule") pour acquérir, désambiguiser et nettoyer des données. Ces applications participatives sont largement connues sous le nom de plates-formes de crowdsourcing où des amateurs peuvent participer à de véritables projets scientifiques ou commerciaux. Ainsi, des demandeurs sous-traitent des tâches en les proposant sur des plates-formes telles que Amazon MTurk ou Crowdflower. Puis, des participants en ligne sélectionnent et exécutent ces tâches, appelés microtasks, acceptant un micropaiement en retour. Ces plates-formes sont confrontées à des défis tels qu'assurer la qualité des réponses acquises, aider les participants à trouver des tâches pertinentes et intéressantes, tirer parti des compétences expertes parmi la foule, respecter les délais des tâches et promouvoir les participants qui accomplissent le plus de tâches. Cependant, la plupart des plates-formes ne modélisent pas explicitement les compétences des participants, ou se basent simplement sur une description en terme de mots-clés. Dans ce travail, nous proposons de formaliser les compétences des participants au moyen d'une structure hiérarchique, une taxonomie, qui permet naturellement de raisonner sur les compétences (détecter des compétences équivalentes, substituer des participants, ...). Nous montrons comment optimiser la sélection de tâches au moyen de cette taxonomie. Par de nombreuses expériences synthétiques et réelles, nous montrons qu'il existe une amélioration significative de la qualité lorsque l'on considère une structure hiérarchique de compétences au lieu de mots-clés purs. Dans une seconde partie, nous étudions le problème du choix des tâches par les participants. En effet, choisir parmi une interminable liste de tâches possibles peut s'avérer difficile et prend beaucoup de temps, et s’avère avoir une incidence sur la qualité des réponses. Nous proposons une méthode de réduction du nombre de propositions. L'état de l'art n'utilise ni une taxonomie ni des méthodes de classement. Nous proposons un nouveau modèle de classement qui tient compte de la diversité des compétences du participant et l'urgence de la tâche. À notre connaissance, nous sommes les premiers à combiner les échéances des tâches en une métrique d'urgence avec la proposition de tâches pour le crowdsourcing. Des expériences synthétiques et réelles montre que nous pouvons respecter les délais, obtenir des réponses de haute qualité, garder l'intérêt des participants tout en leur donnant un choix de tâches ciblé

Kreditderivate : Rückblick und Vorschau nach 13 Jahren der Marktentwicklung

Keine vorhande